Image processing apparatus and image processing method

ABSTRACT

An image processing apparatus that handles RAW images favorably performs display when shooting and display when reproducing. The apparatus images a subject and generates RAW images, and reduces the RAW images and generates reduced RAW images, The RAW images and reduced RAW images are recorded in a recording medium. When shooting, the reduced RAW images are subjected to developing processing for display, and when reproducing, the reduced raw images recorded in the recording medium are acquired and subjected to developing processing in accordance with instructions to reproduce, and each is used for display.

TECHNICAL FIELD

The present invention relates to an image processing apparatus and animage processing method, and more particularly relates to technology ofhandling RAW images of moving images or still images.

BACKGROUND ART

When performing shooting operations, conventional imaging apparatusessubject raw image information imaged by an imaging sensor (RAW images)to de-Bayering processing (demosaicing processing) so as to convert theraw image information into signals made up of luminance and colordifference. The signals are then subjected to developing processing suchas noise removal, optical distortion correction, image optimization, andso forth. The imaging apparatus then generally performs compressionencoding on the developed luminance signals and color differencesignals, and then records in a recording medium.

On the other hand, there are imaging apparatuses capable of recordingRAW images. While the amount of data necessary to record RAW image isgreat, this is preferred by many advanced users. The reason is thatadvantages there are such as correction and deterioration of theoriginal image being minimal, and that post-shooting editing can beperformed.

PTL 1 discloses an imaging apparatus that records RAW images. Disclosedin PTL 1 is a configuration where developing parameters are recordedalong with a RAW image, and when reproducing, the RAW image is developedand reproduced using these developing parameters.

As of recent, imaging sensors in imaging apparatuses have advanced towhere the number of pixels per image is much greater. Further, thenumber of images which can be taken by continuous shooting per second ison the rise. This has led to a compounded increase in the amount ofprocessing for each of the processing making up the developingprocessing, such as de-Bayering processing on RAW images, noise removal,optical distortion correction, and so forth. This has come tonecessitate large-scale circuits and increased electric powerconsumption in the imaging apparatuses, in order to performing real-timedeveloping processing in parallel with shooting. Even then, there may becases where the imaging apparatus cannot exhibit high-level shootingperformance, due to circuits being occupied for developing processing,and constraints related to electric power consumption.

On the other hand, the amount of processing related to developing at thetime of shooting might be reduced by a configuration where RAW imagesare recorded without being developed, such as in PTL 1, but promptlyreproducing and displaying the images becomes difficult since the imagesare recorded in a pre-development state. Further, the fact that RAWimages have peculiarities unique to this data format, and that theformat may differ from one manufacturer to another, can result in RAWimages taken with one device not being able to be reproduced (developed)on another device. Accordingly, conventional RAW image recording formatshave in cases been disadvantageous with regard to ease of use by theuser.

There has been a problem as described above, that in order for animaging apparatus to realize high-level shooting performance and also becapable of fast image output of reproduced images, either expensivecircuits need to be installed for high-output driving, or the RAW imagesneed to be able to be recorded and reproduced in a fast and convenientmanner. Particularly, increased costs are a detriment to the user, so itis important that the imaging apparatus be able to record RAW images inan easy-to-handle manner.

CITATION LIST Patent Literature

PTL 1: Japanese Patent Laid-Open No. 2011-244423

SUMMARY OF INVENTION

Provided is an image processing apparatus and image processing methodenabling an apparatus that records RAW images to suitably performdisplay when shooting and display when reproducing.

An image processing apparatus according to the present inventionincludes: an imaging unit configured to use an imaging sensor to acquirea RAW image representing a pre-development image from imaging signals ofa subject image; a reduction unit configured to reduce the RAW image togenerate a reduced RAW image; a first developing unit configured toacquire the reduced RAW image when shooting, and perform developingprocessing; a recording unit configured to record each data of the RAWimage and the reduced RAW image in a recording medium; an instructingunit configured to instruct reproduction of the RAW image recorded inthe recording medium; a second developing unit configured to acquire thereduced RAW image recorded in the recording medium and performdeveloping processing, in accordance with an instruction forreproduction; and a display processing unit configured to display areproduction image subjected to developing processing by the seconddeveloping unit.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a block diagram illustrating a configuration example of animage processing apparatus according to a first embodiment of thepresent invention.

FIG. 2 is a state (mode) transition diagram according to the embodiment.

FIG. 3 is a flowchart relating to shooting processing according to theembodiment.

FIG. 4A is a diagram illustrating a configuration example of a filerecorded in the embodiment.

FIG. 4B is a diagram illustrating a configuration example of a filerecorded in the embodiment.

FIG. 4C is a diagram illustrating a configuration example of a filerecorded in the embodiment.

FIG. 5 is a flowchart relating to developing processing according to theembodiment.

FIG. 6 is a flowchart relating to reproducing processing according tothe embodiment.

FIG. 7A is a diagram illustrating display processing according to theembodiment.

FIG. 7B is a diagram illustrating display processing according to theembodiment.

FIG. 7C is a diagram illustrating display processing according to theembodiment.

FIG. 8 is a block diagram illustrating a configuration example of anediting apparatus (external apparatus).

FIG. 9A is a flowchart relating to editing processing according to theembodiment.

FIG. 9B is a flowchart relating to execution processing according toembodiment.

FIG. 10 is a block diagram illustrating a configuration example of animage processing apparatus according to a second embodiment of thepresent invention.

FIG. 11 is an explanatory diagram of a pixel array in a RAW image.

DESCRIPTION OF EMBODIMENTS First Embodiment

Embodiments of the present invention will be described in detail withreference to the drawings. FIG. 1 is a block diagram illustrating aconfiguration example of an image processing apparatus according to afirst embodiment of the present invention. An imaging apparatus 100 willbe exemplarily illustrated as the image processing apparatus accordingto the present embodiment in FIG. 1. The imaging apparatus 100 not onlyrecords image information obtained by imaging a subject in a recordingmedium, but also has functions of reproducing image information from arecording medium, performing developing processing, and displaying, andfunctions of exchanging image information with an external apparatus,server (cloud), or the like, and performing editing. Accordingly, theimage processing apparatus according to the present embodiment is notrestricted to being expressed as being an imaging apparatus, and mayalso referred to as being a recording apparatus, reproducing apparatus,recording/reproducing apparatus, communication apparatus, editingapparatus, image processing system, editing system, and so forth.

A control unit 161 in FIG. 1 includes a central processing unit (CPU)and memory (omitted from illustration) storing a control program whichthe CPU executes, thereby controlling the overall processing of theimaging apparatus 100. An operating unit 162 includes input devices suchas keys, buttons, a touch panel, and so forth, for the user to giveinstructions to the imaging apparatus 100. Operation signals from theoperating unit 162 are detected by the control unit 161, and thecomponents are controlled by the control unit 161 so that actions areexecuted corresponding to the operations. A display unit 123 includes aliquid crystal display (LCD) or the like to display images that havebeen shot or reproduced, menu screens, various types of information, andso forth, on the imaging apparatus 100.

In the present embodiment, the terms “shoot” and “shooting” mean theactions of imaging a subject and displaying the image obtained by theimaging on the display unit 123, and also further recording the image ina predetermined recording medium as a file.

Upon starting of shooting action being instructed by the operating unit162, an optical image of a subject to be imaged is input via an opticalunit 101 including a lens unit, and is imaged on an imaging sensor unit102. When shooting, the actions of the optical unit 101 and the imagingsensor unit 102 are controlled by a camera control unit 104, based onevaluation value calculation results such as aperture, focus, shaking,and so forth, that are acquired by an evaluation value calculation unit105, and subject information such as facial recognition resultsextracted by a recognition unit 131.

The imaging sensor unit 102 converts light (subject image) that haspassed through a mosaic color filter of red, green, and blue (ROB)placed at each pixel, into electric signals. The resolution of theimaging sensor is equivalent to, for example, 4K (8 million pixels ormore) or 8K (33 million pixels or more). FIG. 11 is a diagramillustrating an example of a color filter disposed at the imaging sensorunit 102, showing a pixel array of an image which the imaging apparatus100 handles. Red (R), green (G), and blue (B) are arrayed in mosaicfashion at each pixel, with one set of one red pixel, one blue pixel,and two green pixels being orderly arrayed for every four pixels (twopixels by two pixels), as illustrated in FIG. 11. Electric signalsconverted by the imaging sensor unit 102 have the red (R), green (G),and blue (B) components. Note that green (G) can be handled as two typeof green (G0, G1) components at different positions. This sort of pixelarray is generally referred to as a Bayer array. The imaging sensor maybe a charge-coupled device (CCD) image sensor, complementary metal oxidesemiconductor (CMOS) image sensor, or the like. While an RUB color Bayerarray has been illustrated, other arrays may also be optionally used forthe colors and array for the color filter.

Electric signals converted by the imaging sensor unit 102 are handled asimaging signals of individual pixels. A sensor signal processing unit103 subjects the pixels included in the imaging signals to restorationprocessing. This restoration processing involves processing of handlingvalues of missing or unreliable pixels at the imaging sensor unit 102 byperforming interpolation using nearby pixel values for such pixels to berestored, and subtracting a predetermined offset value. This may bealtered so that part or all of this restoration processing is notperformed here but later when developing. In the present embodiment, allimages which have not been subjected to substantive developing arehandled as RAW images, indicating pre-development images, regardless ofwhether or not restoration processing has been performed.

That is to say, image information output from the sensor signalprocessing unit 103 is referred to as RAW image information(hereinafter, “RAW image”), meaning a raw (pre-development) image in thepresent embodiment. The RAW image is supplied to the RAW compressionunit 113 and compressed for efficient recording. The size of the RAWimage is also reduced by a RAW reduction unit 109 for efficient displayand reproduction. The RAW reduction unit 109 resizes the input RAWimages to high-definition (HD) size (equivalent to around 2 millionpixels), for example. Hereinafter, a RAW image which has been reduced bythe RAW reduction unit 109 will be referred to as a reduced RAW image.

The reduced RAW image is supplied to a live view developing unit 111within a developing unit 110, and subjected to developing processing tobe used for display when shooting (live view). The reduced RAW image isalso supplied to a RAW compression unit 113 for use in simplereproduction or as a proxy when editing. The reduced RAW image isrecorded after being compressed by the RAW compression unit 113 forefficient recording, in the same way as with a RAW image. Further, thereduced RAW image is also supplied to the evaluation value calculationunit 105.

The developing unit 110 will be described in detail now. The developingunit 110 has the live view developing unit 111 (first developing unit)that performs developing for the live view when shooting, ahigh-image-quality developing unit 112 (second developing unit) thatperforms high-quality developing when not shooting, and a switch unit121 that selects the output thereof. The live view developing unit 111has the capability to perform real-time developing of the reduced RAWimage in parallel with shooting, while the high-image-quality developingunit 112 has the capability to develop the RAW image before reduction,which is a larger image than a reduced RAW image, in high definition innon-real-time. in the present embodiment, live view developing may alsobe referred to as simple developing or display developing, andhigh-quality developing may also be referred to as main developing orreproduction developing, performed at a requested timing.

The high-image-quality developing unit 112 performs de-Bayeringprocessing (demosaicing processing) on input RAW images or reduced RAWimages, where color interpolation processing is performed to convertinto luminance and color difference (or primary color) signals, removesnoise included in the signals, corrects optical distortion, andoptimizes the image, i.e., performs so-called developing processing.Further, the live view developing unit 111 performs de-Bayeringprocessing (demosaicing processing), i.e., color interpolationprocessing, on reduced RAW images, to convert into luminance and colordifference (or primary color) signals, removes noise included in thesignals, corrects optical distortion, and optimizes the image, i.e.,performs so-called developing processing.

The high-image-quality developing unit 112 performs each process inhigher precision than the live view developing unit 111. Due to the highprecision, a higher quality developed image is obtained as compared tothe live view developing unit 111, but the processing load is greater.Accordingly, the high-image-quality developing unit 112 according to thepresent embodiment is of a configuration where the developing processingcan be performed when reproducing, or when idle after shooting. Thecircuit scale and the maximum (peak) electric power consumption can besuppressed by performing high-image-quality developing after shooting orwhen reproducing, and not while shooting. On the other hand, the liveview developing unit 111 is configured such that the amount ofprocessing involved in developing is less than in the high-image-qualitydeveloping so that developing processing can be performed at high speedwhile shooting, although the image quality is lower than that of thehigh-image-quality developing unit 112. The processing load of the liveview developing unit 111 is low, and accordingly real-time developing ofeach frame can be performed in parallel with shooting actions whenperforming live view shooting of moving images or still images.

The switch unit 121 is switched by the control unit 161, followingcontrol in accordance with operations instructed by the user from theoperating unit 162 or an operating mode currently being executed. Anarrangement may be made where just the one of the live view developingunit 111 and high-image-quality developing unit 112 that is to outputsignals performs developing operations in conjunction with the switchingof the switch unit 121, while the operations of the other are stopped.Although the live view developing unit 111 and the high-image-qualitydeveloping unit 112 are illustrated as being separately-existingconfigurations in the developing unit 110 according to the presentembodiment, a single developing unit may selectively perform live viewdeveloping and high-image-quality developing by switching an operationmode.

The image information subjected to developing processing by thedeveloping unit 110 is further subjected to predetermined displayprocessing by a display processing unit 122, and thereafter displayed ata display unit 123. The image information subjected to developingprocessing can also be output to an external display device connected bya video output terminal 124. The video output terminal 124 includesgeneral-purpose interfaces such as High-Definition Multimedia Interface(HDMI, a registered trademark) and serial digital interface (SDI).

When shooting, image information subjected to live view developing bythe developing unit 110 is supplied to the evaluation value calculationunit 105. The evaluation value calculation unit 105 calculatesevaluation values from the reduced RAW image or developing-processedimage information, such as focus state, exposure state, and so forth.The calculated evaluation values are output to the camera control unit104. Display information indicating the evaluation results is output tothe display processing unit 122. Further, the image informationsubjected to the live view developing is also supplied to therecognition unit 131. The recognition unit 131 has functions ofdetecting and recognizing subject information, such as faces, people,and so forth, in the image information. For example, the recognitionunit 131 detects whether or not there are faces in the screen of theimage information, and if there are faces, outputs informationindicating the position of the faces to the camera control unit 104, andfurther recognizes particular people based on feature information suchas faces and so forth. Display information indicating the detection andrecognition results is output to the display processing unit 122.

Image information subjected to high-image-quality developing by thedeveloping unit 110 is supplied to a still image compression unit 141 ora moving image compression unit 142. In a case of compressing the imageinformation as a still image, the still image compression unit 141 isused. In a case of compressing the image information as a moving image,the moving image compression unit 142 is used. The still imagecompression unit 141 and moving image compression unit 142 performhigh-efficiency encoding (compression coding) of respective imageinformation thereby generating image information where the amount ofinformation has been compressed, and converts into ahigh-image-quality-developed file (still image file or moving imagefile). Examples of standard coding techniques that can be used includeJPEG and the like for still image compression, and MPEG-2, H.264, H.265,and the like for moving image compression.

The RAW compression unit 113 performs high-efficiency coding of the RAWimage data from the sensor signal processing unit 103 and reduced RAWimage data from the RAW reduction unit 109, using technology such aswavelet transform, quantization, entropy coding (differential coding,etc.), and so forth. The RAW compression unit 113 generates a files ofRAW images (RAW files) and reduced RAW images (reduced RAW files) whichhave been compressed by high-efficiency coding. The RAW files andreduced RAW files are first stored in a buffer 115 (storage medium). TheRAW files and reduced RAW files may be left in the buffer 115 and calledup again, or may be stored in the buffer 115 and then transferred to aseparate recording medium and. recorded (deleted from the buffer 115),as described later.

The RAW files and reduced RAW files, and the high-image-qualitydeveloped files (still image files and moving image files) are recordedin a recording medium 152 by a recording/reproducing unit 151. Examplesof the recording medium 152 include a built-in large-capacitysemiconductor memory or hard disk, a detachable memory card, or thelike. The recording/reproducing unit 151 may also read out various typesof files from the recording medium 152 in accordance with useroperations.

The recording/reproducing unit 151 can exchange various types of tilesand related information with an external storage server connected over anetwork via a communication unit 153, or with a mobile informationterminal or personal computer (PC) or the like. The communication unit153 is configured so as to be able to access external apparatuses overthe Internet by wireless communication or cabled communication, or bydirect communication between devices using a communication terminal 154.Accordingly, the recording/reproducing unit 151 can record the RAW filesand reduced RAW files, and the high-image-quality developed files (stillimage files and moving image files) in external apparatuses, and giveinstructions to generate editing information. based on the recordedinformation.

When performing reproducing operations of the imaging apparatus 100, therecording/reproducing unit 151 obtains a desired file from the recordingmedium 152, or from an external apparatus via the communication unit153, and reproduces the file. If the file to be reproduced is a RAW fileor reduced RAW file, the recording/reproducing unit 151 stores theobtained RAW file in the buffer unit 115. If the file to be reproducedis a still image file, the recording/reproducing unit 151 supplies theobtained file to a still image decompression unit 143. If the file to bereproduced is a moving image file, the recording/reproducing unit 151supplies the obtained file to a moving image decompression unit 144.

A RAW decompression unit 114 reads out a RAW file or reduced RAW filestored in the buffer 115, performs conversion opposite to that performedduring compression, and thus decompresses the compressed file. The RAWimage or reduced RAW image decompressed by the RAW decompression unit114 is supplied to the high-image-quality developing unit 112, and issubjected to the high-image-quality developing processing as describedabove.

The still image decompression unit 143 decodes and decompresses stillimage files that have been input, and supplies these to the displayprocessing unit 122 as reproduced images of still images. The movingimage decompression unit 144 decodes and decompresses moving image filesthat have been input, and supplies these to the display processing unit122 as reproduced images of moving images.

Next, operating modes of the imaging apparatus 100 according to thepresent embodiment will be describe with reference to the drawings. FIG.2 is a state (mode) transition diagram according to the presentembodiment. Such mode transition is executed in accordance with useroperation instructions from the operating unit 162, or by determinationby the control unit 161. Accordingly, transition may be performedmanually by operations, or transition may take place automatic. As canbe seen in FIG. 2, the imaging apparatus 100 operations by switchingthrough an idle state (200) to the five modes of a still image shootingmode (201), a still image reproducing mode (202), a moving imageshooting mode (203), a moving image reproducing mode (204), and anediting mode (205), as appropriate.

Next, the operations relating to the still image shooting mode andmoving image shooting mode of the imaging apparatus 100 will bedescribed. FIG. 3 illustrates a flowchart relating to shootingprocessing according to the present embodiment. The flowchart in FIG. 3illustrates processing procedures executed by the control unit 161controlling each processing block. These processing procedures arerealized by loading a program stored in memory (read only memory (ROM))of the control unit 161 to random access memory (RAM), and the CPUexecuting the program.

Upon shooting processing of a still image or moving image starting inFIG. 3, in S301 the control unit 161 determines whether or not to stopshooting. If determination is made to stop the shooting processing,transition is made to the idle state, otherwise, the flow advances toS302. An arrangement may be made where the state transitions to the idlestate even in the shooting mode, of there is no operating input acceptedfor a predated amount of time, or there is time till the next shooting.

In S302, the camera control unit 104 controls the operations of theoptical unit 101 and imaging sensor unit 102 so that shooting isperformed under suitable conditions. For example, lenses included in theoptical unit 101 are moved according to zoom or focusing instructionsgiven by the user, a readout region of the imaging sensor unit 102 isset following instructions of the number of shooting pixels, and soforth. Also, control such as focus adjustment and tracking of aparticular subject is performed based on evaluation value informationand subject information supplied from the later-described evaluationvalue calculation unit 105 and recognition unit 131. Further, in S302,electric signals obtained by conversion at the imaging sensor unit 102are subjected to signal processing at the sensor signal processing unit103 for restoration of pixels. That is to say, the sensor signalprocessing unit 103 subjects missing or unreliable pixels tointerpolation using nearby pixel values for such pixels to be restored,subtracting a predetermined offset value, or the like. In the presentembodiment, image information which is output from the sensor signalprocessing unit 103 after the processing in S302 has ended is called aRAW image, meaning a raw (pre-development) image.

in S303, the RAW reduction unit 109 generates a reduced RAW image fromthe above-described RAW image. In S304, the live view developing unit111 performs developing processing (live view developing) of the reducedRAW image. The control unit 161 switches the switch unit 121 within thedeveloping unit 110 to select output of the image information subjectedto developing processing by the live view developing unit 111.

The live view developing unit 111 subject the reduced RAW image tode-Bayering processing (demosaicing processing), i.e., colorinterpolation processing, so as to convert into signals made up ofluminance and color difference (or primary colors). The signals are thensubjected to developing processing such as noise removal, opticaldistortion correction, image optimization, and so forth. The live viewdeveloping unit 111 realizes high speed of developing and simpleprocessing, by eliminating or restricting the range of the noise removaland optical distortion correction. The developing processing (simpledeveloping) which the live view developing unit 111 performs will bedescribed. Due to the live view developing unit 111 handling reduced RAWimages, and restricting part of the developing processing functions, theimaging apparatus 100 can realize shooting with performance such as 60frames of 2 million pixels per second, for example, using a smallcircuit scale and low power consumption.

The image information subjected to developing processing at the liveview developing unit 111 is supplied to the evaluation value calculationunit 105. In S305, the evaluation value calculation unit 105 computesevaluation values such as focus state, exposure state, and so forth,from the luminance values, contrast values, and so forth, included inthe image information. Note that the evaluation value calculation unit105 may obtain a reduced RAW image before live view developing, andcalculate sonic sort of evaluation value from the reduced RAW image aswell.

The image information subjected to developing processing at the liveview developing unit 111 is also supplied to the recognition unit 131.In S306, the recognition unit 131 performs detection of a subject (suchas a face) from the image information, and recognizes the subjectinformation. For example, the recognition unit 131 detects whether ornot there are faces in the screen of the image information, and if thereare faces, recognizes position of the faces and particular people, andoutputs the results as information.

The image information subjected to developing processing at the liveview developing unit 111 is also supplied to the display processing unit122. In S307, the display processing unit 122 forms a display image fromthe acquired image information, and outputs to the display unit 123 oran external display device for display. A display image on the displayunit 123 is used as a live view display for aiding the user in suitablyframing the subject, i.e., a live view image for shooting. Note that thedisplay image from the display processing unit 122 may be displayed onanother external display device such as a television or the like, viathe video output terminal 124. Further, the display processing unit 122may use the evaluation value information and subject information outputfrom the evaluation value calculation unit 105 and recognition unit 131to mark a focused region on the displayed image, displaying a frame atthe position where a face has been recognized, and so forth, forexample.

In S308, the control unit 161 determines whether or not there has been ashooting instruction from the user (a recording instruction in the caseof a moving image), and if there has been such an instruction, the flowadvances to S310. If there is no instruction. in S308, the flow returnsto S301, and repeats shooting (recording) preparation operations andlive view display.

In S310, the RAW compression unit 113 performs high-efficiency coding(reduced RAW compression) on the reduced RAW image corresponding to theimage to be shot (in the case of a moving image, multiple consecutiveframes) in response to the aforementioned shooting instruction, andgenerates a reduced RAW file. Further, in S311, in response to theabove-described shooting instructions, the RAW compression unit 113performs high-efficiency coding (RAW compression) on the RAW imagecorresponding to the image to be shot (in the case of a moving image,multiple consecutive frames), and generates a RAW file, Thehigh-efficiency coding which the RAW compression unit 113 performs hereis according to known technology such as wavelet transform, entropyencoding, and so forth, but may be lossy coding or lossless coding. Inthe present embodiment, a RAW file is generated which can be restored asa high-image-quality file where the quality is the original RAW image isnot markedly lost even if the RAW is compressed.

In S312, the reduced RAW file is recorded in the buffer 115. Further, inS313, the

RAW file is recorded in the buffer 115. In the case of either file, oncethe file is recorded in the buffer 115, the timing of recording in arecording medium downstream by the recording/reproducing unit 151 may beeither at that point or later. Once the reduced RAW file and the RAWfile are recorded to at least the buffer 115, the flow returns to S301.

Accordingly, the imaging apparatus 100 according to the presentembodiment generates RAW files in response to shooting instructions ofstill images or moving images (recording instructions) at the time ofshooting. Otherwise when shooting, a shooting standby state is effectedwhere an image obtained by live view developing is displayed. A RAW fileis a high-image-quality file so that the image information supplied fromthe sensor signal processing unit 103 is not markedly lost, butdeveloping processing is not needed to generate this file. Accordingly,RAW files can be recorded when shooting with a greater number of imagepixels and faster consecutive shooting speed, using a small-scalecircuit with low electric power consumption.

Next, the structure of various types of files according to the presentembodiment will be described, FIGS. 4A through 4C illustrateconfiguration examples of files recorded in the present embodiment, withFIG. 4A illustrating a reduced RAW file, FIG. 4B a RAW file, and FIG. 4Ca high-image-quality developed file.

The reduced RAW file 400 illustrated in FIG. 4A is recorded in apredetermined recording area of the recording medium 152, for example,by the recording/reproducing unit 151. The reduced RAW file 400 includesa header portion 401, a metadata portion 402, and a compressed dataportion 403. The header portion 401 contains identification codeindicating that this file is of a reduced RAW file format, and so forth,The compressed data portion 403 contains compressed data of a reducedRAW image that has been subjected to high-efficiency coding. In a caseof a reduced RAW file of a moving image, compressed audio data is alsoincluded.

The metadata portion 402 includes identification information 404 such asfile name of corresponding RAW file, and so forth, generated at the sametime as this reduced RAW file. In a case where there is ahigh-image-quality developed file obtained by the corresponding RAW filehaving been subjected to high-image-quality developing, thisidentification information 407 is stored. Also, if this reduced RAWimage has already been developed, information 405 of the developingstatus thereof is included. Further, shooting metadata 406 includingevaluation values and subject information calculated and detected at theevaluation value calculation unit 105 and recognition unit 131 at thetime of shooting, and information from the optical unit 101 and imagingsensor unit 102 at the time of shooting (e.g., lens type identificationinformation, sensor type identification information, etc.), is included.Although omitted from illustration, identification code of a recordingmedium where a RAW file generated at the same time is recorded, pathinformation of a folder where recorded, a thumbnail of the image, and soforth, may further be included.

The RAW file 410 illustrated in FIG. 4B is recorded in a predeterminedrecording area of the recording medium 152, for example, by therecording/reproducing unit 151. The RAW file 410 includes a headerportion 411, a metadata portion 412, and a compressed data portion 413.The header portion 411 contains identification code indicating that thisfile is of a RAW file format, and so forth. The compressed data portion413 contains compressed data of a RAW image that has been subjected tohigh-efficiency coding. In a case of a RAW file of a moving image,compressed audio data is also included.

The metadata portion 412 includes identification information 414 such asfile name of corresponding reduced RAW file, and so forth, generated atthe same time as this RAW file. In a case where there is ahigh-image-quality developed file obtained by this RAW image having beensubjected to high-image-quality developing, this identificationinformation 417 is stored. Also, information 415 of the developingstatus thereof in the high-image-quality developing is included.Further, shooting metadata 416 including evaluation values and subjectinformation calculated and detected at the evaluation value calculationunit 105 and recognition unit 131 at the time of shooting, andinformation from the optical unit 101 and imaging sensor unit 102 at thetime of shooting (e.g., lens type identification information, sensortype identification information, etc.), is included. Although omittedfrom illustration, identification code of a recording medium where areduced RAW file generated at the same time is recorded, pathinformation of a folder where recorded, a thumbnail of the image, and soforth, may further be included. Alternatively, the actual data of thereduced RAW file generated at the same time as this RAW file itself maybe made into metadata and sorted in the metadata portion 412. Further,the actual data of high-image-quality developed file corresponding tothis RAW file may itself be made into metadata and stored in themetadata portion 412.

The high-image-quality developed file 420 illustrated in FIG. 4C isrecorded in a predetermined recording area of the recording medium 152,for example, by the recording/reproducing unit 151. Thehigh-image-quality developed file 420 includes a header portion 421, ametadata portion 422, and a compressed data portion 423. The headerportion 421 contains identification code indicating that this file is ofa high-image-quality developed file format, and so forth. The compresseddata portion 423 contains compressed data of still image portions andmoving image portions of a high-image-quality developed file. In a caseof a moving image, compressed audio data is also included.

The metadata portion 422 includes identification information 424 such asthe file name of a reduced RAW file corresponding to thishigh-image-quality developed file, and so forth. Also, identificationinformation 427 of the file name of a RAW file corresponding to thehigh-image-quality developed file is included. Also, information 425 ofthe developing status of this high-image-quality developed file in thehigh-image-quality developing is included. Further, shooting metadata426 including evaluation values and subject information calculated anddetected at the evaluation value calculation unit 105 and recognitionunit 131 at the time of shooting, and information from the optical unit101 and imaging sensor unit 102 at the time of shooting (e.g., lens typeidentification information, sensor type identification information,etc.), is included. Although omitted from illustration, identificationcode of a recording medium where the corresponding RAW file and reducedRAW file is recorded, path information of a folder where recorded, athumbnail of the image, and so forth, may further be included.

The above-described file structures according to the present embodimentare only exemplary, and may have structures in common with standardssuch as Design rule for Camera File system (DCF), Exchangeable ImageFile format (EXIF), Advanced Video Codec High Definition (AVCHD), orMaterial eXchange Format (MXF).

An example of high-image-quality developing processing of the imagingapparatus 100 will be described. FIG. 5 is a flowchart relating todeveloping processing according to the present embodiment. The flowchartin FIG. 5 illustrates the processing procedures which the control unit161 carries out by controlling the processing blocks, and is realized byloading a program stored in memory (ROM) of the control unit 161 tomemory (RAM), and the CPU executing the program.

In FIG. 5, the control unit 161 determines whether or not to perform“catch-up developing” according to user settings when in an idle state(S501), and if determination is made not to perform catch-up developing,the flow ends (returns). If determination is made to perform catch-updeveloping, the flow advances to S520.

The “catch-up developing” according to the present embodiment means thatafter the shooting operation has ended, a RAW file recorded in thebuffer 115 or recording medium 152 is read out, and the RAW image issubjected to high-image-quality developing, thereby generating ahigh-image-quality developed file. This catch-up developing isdeveloping processing performed when the device is idle or in thebackground of other processing. The name conies from developingprocessing seeming to chase a RAW file recorded earlier, as if it weretrying to catch up. While both still images and moving images may beincluded in RAW files which are the object of catch-up developingaccording to the present embodiment, an example of still images will bedescribed below.

As described above, a recorded RAW file is a high-image-quality filewhere the image information supplied from the sensor signal processingunit 103 is not markedly lost, but reproduction display or printingcannot be performed immediately since the RAW file is data beforedeveloping, and a request for reproduction display or printingnecessities time for RAW developing. Also, a RAW file is not a format inwidespread use like JPEG or the like, so reproducing environments whichcan handle RAW files are restricted. Accordingly, the catch-updeveloping according to the present embodiment is a useful function.Upon catch-up developing being performed in the present embodiment, aRAW file that has already been recorded is read out, subjected tohigh-image-quality developing processing by the high-image-qualitydeveloping unit 112, and the generated high-image-quality developedstill image file is recorded in the recording medium 152 or the like.This sort of catch-up developing is performed in states where theprocessing load of the apparatus is relatively low in standby for useroperations (when idle), such as in between shootings, when in standby inreproducing mode, in sleep state, or the like. The catch-up developingmay be initiated manually, but preferably is designed so that thecontrol unit 161 automatically executes in the background in a certainstate. Due to this configuration, even in a case where there is arequest at a later time for reproduction of high-image-quality images,such as displaying to confirm details of an image or printing or thelike, there is no delay in the developing processing (reproductionoutput) each time, and a general operating environment the same as withconventional still image files can be used.

In S520 in FIG. 5, the control unit 161 determines whether the catch-updeveloping for the RAW file of interest is already processes orunprocessed. Examples of conceivable ways of determining include whetheror not a high-image-quality developed file has been created, fromidentification information in the RAW file 410, or the like.Alternatively, the developing status information 415 in the RAW file 410may be referenced to make this determination in the same way.Alternatively again, a table file indicating the state of execution ofdeveloping processing of a series of RAW files may be preparedseparately and used for determination.

If the control unit 161 determines that catch-up developing has beencompeted for all RAW files of interest, the flow ends (returns) there,and the imaging apparatus 100 transitions to the idle state. If thereare any RAW files regarding which the catch-up developing isunprocessed, the flow advances to S521. If a RAW file regarding whichcatch-up developing is unprocessed has been buffered in the buffer 115in S521, the flow advances to S523. If not, the RAW file is read outfrom the recording medium 152 or the like in S522, and temporarily heldat the buffer 115.

The data of the buffer 115 is updated so that the images which are newerin the order of shooting are held with higher priority. That is to say,images shot in the past are deleted from the buffer in order. Thus, themost recently shot images are always held in the buffer, so S522 can beskipped and processing performed at high speed. Further, an arrangementwhere catch-up developing is performed starting from an image shotimmediately prior and going back in time enables images held in thebuffer to complete processing with higher priority, so the processing ismore efficient.

In S523, the RAW decompression unit 114 decompresses the RAW file readout from the buffer 115 or recording medium 152 and buffered, therebyrestoring the RAW image. In S524, the high-image-quality developing unit112 executes high-image-quality developing processing on the restoredRAW image, and outputs the high-image-quality developed image to thedisplay processing unit 122 and still image compression unit 141 via theswitch unit 121. At this time, a display image may be displayed on thedisplay unit 123 if the imaging apparatus 100 is in a state where imagesdeveloped later can be displayed.

The high-image-quality developing unit 112 performs de-Layeringprocessing (demosaicing processing) on the RAW images, i.e., performscolor interpolation processing to convert into signals made up ofluminance and color difference (or primary color) signals, removes noiseincluded in the signals, corrects optical distortion, and optimizes theimage, i.e., performs so-called developing processing. The size (numberof pixels) of the developed image generated at the high-image-qualitydeveloping unit 112 is the full-size read out from the imaging sensorunit 102, or a size set by the user, so the image quality is much higherthan the live view developing image that handles around 2 millionpixels. The high-image-quality developing unit 112 performs each processin higher precision than the live view developing unit 111, so a higherquality developed image is obtained as compared to the live viewdeveloping unit 111, but the processing load is greater. Thehigh-image-quality developing unit 112 according to the presentembodiment has a configuration with increase in circuit scale andelectric power consumption suppressed, by avoiding real-time developingprocessing in parallel with shooting, and enabling developing to beperformed taking time.

The image information subjected to developing processing at thehigh-image-quality developing unit 112 is supplied to the still imagecompression unit 141 or the moving image compression unit 142. In thecase of a still image, the still image compression unit 141 handlescompression. In S525, the still image compression unit 141 performshigh-efficiency coding processing (still image compression) on theacquired high-image-quality developed image, thus generating ahigh-image-quality developed file (still image file). Note that thestill image compression unit 141 performs compression processing by aknown format such as JPEG or the like. In S526, therecording/reproducing unit 151 records the high-image-quality developedfile in the recording medium 152 or the like.

If the control unit 161 determines in S527 that the imaging apparatus100 is no longer in the idle state, partway through the flow, transitionis made to interruption processing. Otherwise, the flow returns to S520.In the processing of S520 and thereafter, if there is a RAW imageregarding which catch-up developing is unprocessed, the same processingcan be repeatedly executed for each image. On the other hand, in a caseof interrupting the catch-up processing in S528, information at thepoint of the control unit 161 interrupting (RAW file to be interrupted,identification information regarding whether developing completed or notcompleted, etc.) is stored in memory or the recording medium 152 asrecovery information (interruption processing). At the time ofperforming catch-up developing, the control unit 161 references thisrecovery information so as to resume from the RAW file that wasinterrupted. After the interruption processing ends, the imagingapparatus 100 returns to the idle state.

The high-image-quality developed file recorded in S526 is of a filestructure such as illustrated in FIG. 4C. The information such as thefile name of the original RAW file for this high-image-quality developedfile is written to the metadata portion 422. Information 425 is alsodescribed regarding the fact that this high-image-quality developed filehas been subjected to high-image-quality developing by thehigh-image-quality developing unit 112, and the developing statusindicating the content of this developing. Shooting metadata 426including evaluation values and subject information calculated anddetected by the evaluation value calculation unit 105 and recognitionunit 131, and information at the time of shooting from the optical unit101 and imaging sensor unit 102, extracted from the metadata of theoriginal RAW file, is also copied. Further, the recording/reproducingunit 151 updates information for each metadata portion of the originalRAW file for the catch-up developing, and the reduced RAW file thereofas well, to the newest information relating to the generatedhigh-image-quality developed file.

An arrangement where the recording/reproducing unit 151 records the newhigh-image-quality developed file recorded in S526 after thehigh-image-quality developing, using a file name similar or correlatedwith the original RAW file, facilitates identification. For example, afile name where only part of the file name has been changed (e.g.,suffix or ending characters) while the rest of the file name is thesame, is desirable.

Thus, the imaging apparatus 100 according to the present embodimentexecutes catch-up developing in states where the processing load of theapparatus is relatively low in standby for user operations (when idle),such as in between shootings, when in standby in reproducing mode, insleep state, or the like. High-image-quality developed files aresequentially created from the RAW files. Thus, even when receiving arequest for reproduction of a high-image-quality image, such as forconfirmation display of detailed portions of the image or for printing,there is no delay due to developing processing (reproduction output)each time this occurs, and the files can be used in a general operatingenvironment in the same way as conventional still image files.

Next, operations relating to the still image reproducing mode and movingimage reproducing mode of the imaging apparatus 100 will be described.FIG. 6 is a flowchart relating to reproducing processing of the presentembodiment. The flowchart in FIG. 6 illustrates the processingprocedures which the control unit 161 carries out by controlling theprocessing blocks, and is realized by loading a program stored in memory(ROM) of the control unit 161 to memory (RAM), and the CPU executing theprogram.

In S601 in FIG. 6, the control unit 161 determines whether or not tostop reproduction processing. In a case of stopping reproductionprocessing, the imaging apparatus 100 returns to the idle state.Otherwise, the flow advances to S602.

In S602, the control unit 161 reads out a reduced RAW image file whichis the object of reproduction processing, and determines whether or notto reproduce the reduced RAW image. RAW files are high-resolution, andaccordingly take time to develop as described above. Further, there is apossibility that a high-image-quality developed file has not net beengenerated, so when reproducing, the control unit 161 reproduces RAWimages with higher priority than others. A reduced RAW image has around2 million pixels, so high-speed processing can be performed in live viewdeveloping in the same way, sudden requests for reproducing can beimmediately handled, and a reproduced image can be quickly output inquick response. Note however, that a reduced RAW image has limited imagequality. While this is effective for general confirmation of the image,it may be insufficient for usages such as confirming details of theimage or printing. Accordingly, The imaging apparatus 100 also executesreproduction processing of other images depending on usage, as describedbelow.

In a case of reproducing a reduced RAW image, in S620 the control unit161 reproduces the reduced RAW file to be reproduced, from the buffer115 or recording medium 152 or the like. In S621, the RAW decompressionunit 114 decompresses the compressed reduced RAW image obtained from thereproduced reduced RAW file. Further, in S622 the high-image-qualitydeveloping unit 112 develops the decompressed reduced RAW image togenerate a reproduction image, which is supplied to the displayprocessing unit 122. While developing of the reduced. RAW image has beendescribed as being performed by the high-image-quality developing unit112 in S622, this may be performed by the live view developing unit 111instead.

In a case of not reproducing the reduced RAW image, in S603 the controlunit 161 determines whether or not to reproduce a high-image-qualitydeveloped image. A condition for reproducing a high-image-qualitydeveloped image is that the above-described catch-up developing has beenperformed, or that high-image-quality developing has already beencompleted by reproduction of the RAW file in response to a user request.In a case of reproducing a high-image-quality developed image, in S630the control unit 161 reproduces the high-image-quality developed file tobe reproduced from the recording medium 152 or the like. In S631, thestill image decompression unit 143 or the moving image decompressionunit 144 decompresses the compressed high-image-quality developed imageobtained from the reproduced high-image-quality developed file togenerate a reproduction image (still image or moving image), which issupplied to the display processing unit 122.

In a case where no reduced RAW file is reproduced and nohigh-image-quality developed image is reproduced, the RAW file isreproduced. Examples of usage environments of reproduction of a RAW filein the present embodiment will be described. with reference to FIGS. 7A,7B, and 7C. FIGS. 7A, 7B, and 7C are diagrams illustrating examples ofdisplay processing of the present embodiment. FIGS. 7A, 7B, and 7C areexamples of image displays at different timings.

Display example 700 in FIG. 7A is an example of reduced display of siximages, denoted by reference numeral 701, on the display unit 123 in areduced manner. This display state is a list display. Display example710 in FIG. 7B is an example of display of one image 711 on the entiredisplay unit 123. This display state is a normal display. Displayexample 720 in FIG. 7C is an example of display of an enlarged portionof one image 721 on the entire display unit 123. This display state isan enlarged display. The enlarged display assumes usage where part of asubject image is displayed in an enlarged manner such as in the displayexample 720, to confirm details of an image that has been shot,partially extract (trimming), or check whether in focus or not, forexample.

Resolution is sufficient for display of the reproduced image obtainedfrom the reduced RAW image in the state of display examples 700 or 710.However, resolution of a reduced. RAW image is insufficient in a case ofenlarged display such as in display example 720 (leads to poorresolution), so a RAW file that has high resolution is prefer reproducedand displayed. In a case of performing enlarged display of thereproduced image, the flow transitions to RAW file reproduction in S604and thereafter.

In a case of reproducing a RAW image, in S604 the control unit 161determines whether or not the RAW file to be reproduced has beenbuffered in the buffer 115. In a case where the RAW file has beenbuffered, the flow advances to S606, and if not buffered, to S605. InS605, the control unit 161 reads out the RAW file to be reproduced fromthe recording medium 152 or the like, and buffers in the buffer 115.

The data of the buffer 115 is updated so that the images which are newerin the order of shooting in the shooting mode are held with higherpriority. That is to say, images shot in the past are deleted from thebuffer in order. Thus, the most recently shot images are always held inthe buffer, so S605 can be skipped and processing performed at highspeed.

In S606, the RAW decompression unit 114 obtains the buffered RAW file,and decompresses the compressed RAW image obtained from this RAW file.Further, in S607 the high-image-quality developing unit 112 performshigh-image-quality developing of the decompressed RAW image to generatea reproduction image, which is supplied to the display processing unit122. Note that the imaging apparatus 100 can also newly create ahigh-image-quality developed file corresponding to the RAW file to bereproduced by the high-image-quality developing in S607.

In S608, the display processing unit 122 outputs one of the reproductionimages according to the object of reproduction to the display unit 123for display. The display form is as illustrated in FIGS. 7A through 7C,The display processing unit 122 can also output the display image fromthe video output terminal 124 to an external apparatus. Upon thedisplaying in S608 ending, the flow returns to S601.

It can be assumed that the RAW file reproduction from S604 andthereafter in FIG. 6 may occur at a timing where catch-up developing hasnot the been performed. This conversely means that even in a case ofenlarged display that requires high image quality, thehigh-image-quality developed image can be provided without newlydeveloping the RAW file as long as the high-image-quality developed filehas been created. Catch-up developing is performed in the presentembodiment in states where the processing load of the apparatus isrelatively low in standby for user operations (when idle), such as inbetween shootings, when in standby in reproducing mode, in sleep state,or the like. Also, a high-image-quality developed image can be createdwhen reproducing a RAW file under user instructions. high-image-qualitydeveloped files are sequentially created from RAW files in this way. Themore high-image-quality developing is performed beforehand, the lowerthe frequency of high-image-quality developing occurring at the point ofrequesting an enlarged display, so the faster high-image-quality imagescan be output for enlarged displays, and the better the operability isanticipated to be.

Description has been made above that, in a case where the RAW file isbuffered in the buffer 115, S605 can be skipped, so the image can bedisplayed faster. Accordingly, when performing display of the displayexamples 700 and 710 in FIGS. 7A and 7B, the RAW files corresponding tothe images 701 and 711 are prefer read out from the recording mediumbeforehand and loaded to the buffer 115 beforehand, so that RAW filesare held in the buffer 115 as much as possible. An arrangement where therecording/reproducing, unit 151 reads out the corresponding RAW filefrom the recording medium 152 or the like and buffers before aninstruction to enlarge enables display to be made at an even fasterresponse speed when an instruction for an enlarged display such as inthe display example 720 is made.

Next, operations relating to the editing mode of the imaging apparatus100 will be described. FIG. 8 is a block diagram illustrating aconfiguration example of an editing apparatus (external apparatus) thatperforms editing processing. FIGS. 9A and 9B are flowcharts relating toediting processing and editing execution of the present embodiment. Theflowcharts in FIGS. 9A and 9B illustrate the processing procedures whichthe control unit 161 carries out by controlling the processing blocks,and is realized by loading a program stored in memory (ROM) of thecontrol unit 161 to memory (RAM), and the CPU executing the program.

An editing apparatus 800 in FIG. 8 is an external apparatus from theimaging apparatus 100, more specifically, an external storage or server,or a mobile information terminal (tablet, smartphone, etc.), personalcomputer (PC), or the like. The editing apparatus 800 can exchangevarious types of files and related information with the imagingapparatus 100. A communication unit 802 has a configuration whichenables it to access the imaging apparatus 100 via the Internet or bydirect communication between apparatuses, by wireless or cabledcommunication using a communication terminal 801.

At the time of performing editing processing or beforehand, the editingapparatus 800 acquires a reduced RAW file from the external imagingapparatus 100 that is connected over a network via the communicationunit 802. The received reduced RAW file is saved in a recording medium803 via an internal bus 811. The internal bus 811 is connected with theparts within the editing apparatus 800, and serves as a data bus andsystem bus.

The reduced RAW file is decompressed developed, and image informationobtained thereby is used in the editing processing. A RAW decompressionunit 804 reads out a desired reduced RAW file saved in the recordingmedium 803, and decompresses the compressed reduced RAW image. A RAWdeveloping unit 805 performs developing processing of the decompressedreduced RAW image. The image obtained by the developing processing fromthe reduced RAW image is displayed on a display unit 809.

A control unit 808 includes a CPU, and unshown memory storingapplication programs and control programs which the CPU executes. Thecontrol unit 808 controls the editing processing, which is executedaccepting editing instructions from a user monitoring an image displayedon the display unit 809, under control of an editing application programexecuted by the control unit 808. Editing instructions are input fromthe user by a user interface unit 807. The user interface unit 807 usesan operating unit realized by a touch panel, mouse, keyboard, dedicatedor general-purpose console, or the like for example, and an editingapplication that provides display information.

The displayed reduced RAW image is subjected to trimming, compositing,and so forth, according to editing instructions from the user interfaceunit 807. If the file is a moving image, instructions that are appliedmay also include optional scene selection by cut-in/cut-out beingspecified, effects being applied and so forth. An editing informationgenerating unit 806 generates editing information according to thecontents of editing. The editing information is the contents of editingwhich the image obtained from the reduced RAW image has been subjectedto, in the form of data information. The editing information hasdescribed therein whether or not editing has been applied to each frameof still images or moving images, and the contents of editing. Theediting information may further include actual data of the reduced RAWimage which has been subjected to the editing processing. This editinginformation is recorded in the recording medium 803, and alsotransmitted to the imaging apparatus 100 in response to requests fromthe imaging apparatus 100.

Next, the flows of editing processing and editing execution by theimaging apparatus 100 will be described with reference to FIGS. 9A and9B. FIG. 9A is an editing processing flow, and FIG. 9B is an editingexecution flow. Upon editing processing starting, in S901 of FIG. 9A thecontrol unit 161 determines whether or not to stop editing processing,in a case of stopping editing processing, the imaging apparatus 100returns to the idle state. In a case of continuing editing processing,the flow advances to S902.

In S902, the control unit 161 transmits the reduced RAW filecorresponding to the RAW file to be edited, to the editing apparatus 800which is an external apparatus, as editing data. In S903, the controlunit 161 issues an editing command to the editing apparatus 800. Theediting command is transmitted to the control unit 808, and serves as anediting information generating instruction to the editing apparatus 800.After S903, the flow returns to S901.

Upon receiving the aforementioned editing command, the editing apparatus800 performs the editing processing described above, thereby generatingediting information. The control unit 161 can start the editingexecution flow in FIG. 9B under user instructions, under the conditionthat generating of the editing information has ended at the editingapparatus 800.

Upon editing execution being started, in S911 in FIG. 9B the controlunit 161 determines whether or not to stop editing processing. In a caseof stopping editing processing, the imaging apparatus 100 returns to theidle state. In a case of continuing editing execution, the flow advancesto S912.

In S912, the control unit 161 receives from the editing apparatus 800the editing information corresponding to the RAW file to be edited,using the communication unit 153. In S913, the control unit 161references this editing information to apply editing or reproductionaccording to the content described in the received editing informationas to the RAW file or reduced RAW file within the imaging apparatus 100.That is to say, the contents of editing on the reduced RAW imageperformed at the editing apparatus 800 are reproduced on the originalRAW image or reduced RAW image. within the imaging apparatus 100. AfterS913, the flow returns to S911. The RAW image subjected to editingexecution in this way may be saved as a file where the original file hasbeen updated, or the original image and editing information may each besaved separately and the editing information being reflected whenreproducing the image, i.e., edited, each time the image is reproduced.

In this way, the reduced RAW image is used as editing data or as a proxyfor the RAW image, whereby the process of editing be streamlined andcapabilities increased by this configuration of editing using anexternal apparatus. Moreover, the overall load on the system regardingediting processing can be reduced.

Second Embodiment

FIG. 10 is a block diagram illustrating a configuration example of animage processing apparatus according to a second embodiment of thepresent invention. FIG. 10 illustrates an imaging apparatus 1000 as theimage processing apparatus according to the present embodiment. Theimaging apparatus 1000 not only records image information obtained byimaging a subject in a recording medium, but also has functions ofreproducing image information from a recording medium, performingdeveloping processing, and displaying, and functions of exchanging imageinformation with an external apparatus, server (cloud), or the like, andperforming editing. Accordingly, the image processing apparatusaccording to the present embodiment is not restricted to being expressedas being an imaging apparatus, and may also referred to as being arecording apparatus, reproducing apparatus, recording/reproducingapparatus, communication apparatus, editing apparatus, image processingsystem, editing system, and so forth.

Configurations of the imaging apparatus 1000 according to the presentembodiment which are the same as those of the imaging apparatus 100 inthe first embodiment described above will be denoted with the samereference numerals, and description thereof will be omitted.

In FIG. 10, the imaging apparatus 1000 has an imaging module 170 servingas an imaging unit, and an image processing module 180 serving as animage processing unit, the imaging module 170 and the image processingmodule 180 being connected by a bus 107. The RAW data and the likeobtained by imaging within the imaging module 170 is supplied to theimage processing module 180 via the bus 107. The imaging module 170includes the imaging sensor unit 102, sensor signal processing unit 103,camera control unit 104, and encoding unit 106. The image processingmodule 180 includes a decoding unit 108, RAW reduction unit 109,developing unit 110, live view developing unit 111, high-image-qualitydeveloping unit 112, switch unit 121, RAW compression unit 113, RAWdecompression unit 114, and buffer 115. The image processing module 180further includes the evaluation value calculation unit 105, recognitionunit 131, display processing unit 122, still image compression unit 141,moving image compression unit 142, still image decompression unit 143,moving image decompression unit 144, recording/reproducing unit 151, andcontrol unit 161. The imaging apparatus 1000 also has the optical unit101, display unit 123, video output terminal 124, communication unit153, communication terminal 154, and built-in or detachable recordingmedium 152.

Configurations of the imaging apparatus 1000 according to the presentembodiment which differ from those of the imaging apparatus 100according to the first embodiment described above, will be described indetail. The imaging module 170 internally has the encoding unit 106.This exists to compress RAW data, to reduce transmission of the massiveRAW data from overwhelming the communication band over the bus 107.Examples of encoding processing (compression processing) which can beapplied at the encoding unit 106 include compression by DifferentialPulse Code Modulation (DPCM) and Golomb coding. This method reduces theamount of information of pixels in pixel data subjected to DPCMprocessing by Golomb coding of difference values among the pixel datathereof. Alternatively, compression may be performed where unnecessaryhigh-hand components of the pixel data are deleted using one-dimensionaldiscrete cosine transform (DCT). The rate of compression may be fixedwhen carrying out either method, or may be designed to be adjustableaccording to user instructions or the shooting mode.

Pixel data obtained by the imaging sensor unit 102 converting thesubject image when shooting is subjected to pixel restoration processingby the sensor signal processing unit 103. This restoration processinginvolves processing of handling missing or unreliable pixels at theimaging sensor unit 102 by performing interpolation using nearby pixelvalues for such pixels to be restored, and subtracting a predeterminedoffset value. This may be altered so that part or all of thisrestoration processing is not performed here but later when developing.

The encoding unit 106 performs encoding processing of the pixel datasupplied from the sensor signal processing unit 103 according to anaforementioned method. The pixel data in the state of having beencompressed in the encoding processing is transmitted to the imageprocessing module 180 via the bus 107. The compressed pixel data isdecoded by the decoding unit 108 disposed at the input portion of theimage processing module 180. The decoding unit 108 subjects inversetransform from the encoding unit 106 upstream, to decode the compressedpixel data.

The pixel data that has been compressed by the encoding unit 106 andfurther decoded by the decoding unit 108 is referred to as a RAW image,meaning a raw (pre-development) image in the present embodiment. A RAWimage that has been subjected to compression can still be handled as ahigh-quality RAW image, in the same way as in the first embodiment.

Thereafter, the RAW image is supplied to the RAW compression unit 113and compressed again, for efficient recording. The RAW image is reducedin size by the RAW reduction unit 109 to effectively perform display andreproduction. The RAW reduction unit 109 resizes the input RAW images toHD size (equivalent to around 2 million pixels), for example, therebygenerating a reduced RAW image. Processing of the RAW image and reducedRAW image thereafter is the same as in the first embodiment.

Note that the decoding unit 108 and RAW decompression unit 114 in theimage processing module 180, which both perform decompression/decodingof compressed RAW images may be realized by a circuit where part or allof the processing circuits of the two are shared or integrated.

The shooting processing, developing processing, reproduction processing,editing processing, and editing executing processing performed at theimaging apparatus 1000 according to the present embodiment, and theconfigurations of the files created thereby, are the same as in thefirst embodiment.

Thus according to the present embodiment, the same functions as thefirst embodiment can be realized, while improving the transmissionefficiency of RAW data within the imaging apparatus 1000.

While the first and second embodiments have been described above, thepresent invention is not restricted to these embodiments, rather,various modifications may be made by circuit arrangements as suitablewithout departing from the technical idea of the present invention.

Other Embodiments

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application Nos.2014-186871, 2014-186872, and 2014-186873, all filed Sep. 12, 2014,which are hereby incorporated by reference herein in their entirety.

1. An image processing apparatus comprising: an imaging unit configuredto use an imaging sensor to acquire a RAW image representing apre-development image from imaging signals of a subject image; areduction unit configured to reduce the RAW image to generate a reducedRAW image; a first developing unit configured to acquire the reduced RAWimage when shooting, and perform developing processing; a recording unitconfigured to record each data of the RAW image and the reduced RAWimage in a recording medium; an instructing unit configured to instructreproduction of the RAW image recorded in the recording medium; a seconddeveloping unit configured to acquire the reduced RAW image recorded inthe recording medium and perform developing processing, in accordancewith an instruction for reproduction; and a display processing unitconfigured to display a reproduction image subjected to developingprocessing by the second developing unit.
 2. The image processingapparatus according to claim 1, wherein the first developing unitperforms display developing where an image that has been shot issubjected to developing processing in real time, and wherein the seconddeveloping unit performs reproduction developing which is of higherquality that of the first developing unit, to reproduce the recordedimage.
 3. The image processing apparatus according to claim 1, furthercomprising: a control unit configured to, in response to the instructionfor reproduction, reproduce a developed image of the RAW image in a casewhere the RAW image recorded in the recording medium has been developed,and develop and display the reduced RAW image by the second developingunit in a case where the RAW image recorded in the recording medium hasnot been developed.
 4. The image processing apparatus according to claim1, wherein the second developing unit acquires the RAW image recorded inthe recording medium and performs developing processing, in accordancewith predetermined conditions, and wherein the recording unit generatesa high-image-quality developed image file from the developed image ofthe RAW file generated by the second developing unit, and records thegenerated high-image-quality developed image file in the recordingmedium.
 5. The image processing apparatus according to claim 1, furthercomprising: an encoding unit configured to perform encoding to compressthe amount of information regarding data of pixels in the imagingsignals; a transmission unit configured to transmit the encoded pixeldata via a bus; a decoding unit configured to receive and decode thetransmitted pixel data; and a compression unit configured to restore theRAW image from the decoded pixel data, and perform RAW imagecompression.
 6. An image processing method comprising: imaging, using animaging sensor, to acquire a RAW image representing a pre-developmentimage from imaging signals of a subject image; reducing the RAW image togenerate a reduced RAW image; first developing to acquire the reducedRAW image when shooting, and perform developing processing; recordingeach data of the RAW image and the reduced RAW image in a recordingmedium; instructing reproduction of the RAW image recorded in therecording medium; second developing to acquire the reduced RAW imagerecorded in the recording medium and perform developing processing, inaccordance with an instruction for reproduction; and displaying areproduction image subjected to developing processing in the seconddeveloping.
 7. A non-transitory computer-readable storage medium storinga program for causing a computer to function the each unit of the imageprocessing apparatus according to claim 1,
 8. An image processingapparatus comprising: an imaging unit configured to use an imagingsensor to acquire a RAW image representing a pre-development image fromimaging signals of a subject image; a reduction unit configured toreduce the RAW image to generate a reduced RAW image; a compression unitconfigured to compress each of the RAW image and the reduced RAW image;a recording unit configured to record each data of the compressed RAWimage and the compressed reduced RAW image in a recording medium; adeveloping unit configured to read out the compressed. RAW imagerecorded in the recording medium in a predetermined state, performdeveloping processing in the background, and generate a developed image;and a reproducing unit configured to, at the time of reproducing of acertain RAW image being instructed, perform reproduction output of thedeveloped image in a case where developing processing by the developingunit has ended, and reproduce the reduced RAW image reduced from therecording medium and perform developing processing by the developingunit when reproducing and perform reproduction output in a case wheredeveloping processing by the developing unit has not ended.
 9. The imageprocessing apparatus according to claim 8, wherein the developing unitreads out the compressed RAW image from the recording medium when in anidle state, and performs developing processing.
 10. The image processingapparatus according to claim 8, wherein the developing unit reads outthe RAW image from the recording medium in between shootings orreproductions, and performs developing processing.
 11. The imageprocessing apparatus according to claim 8, wherein the recording unitgenerates an image file from a developed image generated by thedeveloping unit, and records the image file in the recording medium. 12.The image processing apparatus according to claim 8, further comprising:an encoding unit configured to perform encoding to compress the amountof information regarding data of pixels in the imaging signals; atransmission unit configured to transmit the encoded pixel data via abus; and a decoding unit configured to receive and decode thetransmitted pixel data, wherein the compression unit acquires the RAWimage from the decoded pixel data, and performs RAW image compression.13. An image processing method comprising: imaging, using an imagingsensor, to acquire a RAW image representing a pre-development image fromimaging signals of a subject image; reducing the RAW image to generate areduced RAW image; compressing each of the RAW image and the reduced RAWimage; recording each data of the compressed RAW image and thecompressed reduced RAW image in a recording medium; developing to readout the compressed RAW image recorded in the recording medium in apredetermined state, perform developing processing in the background,and generate a developed image; and reproducing to, at the time ofreproducing of a certain RAW image being instructed, performreproduction output of the developed image in a case where developingprocessing has ended, and reproduce the reduced RAW image reduced fromthe recording medium and perform developing processing by the developingunit when reproducing and perform reproduction output in a case wheredeveloping processing has not ended.
 14. A non-transitorycomputer-readable storage medium storing a program for causing acomputer to function the each unit of the image processing apparatusaccording to claim 8.